Hierarchical deep multi-modal network for medical visual question answering

Visual Question Answering in Medical domain (VQA-Med) plays an important role providing medical assistance to the end-users. These users are expected raise either a straightforward question with Yes/No answer or challenging that requires detailed and descriptive answer. The existing techniques VQA-Med fail distinguish between different types sometimes complicates simpler problems, over-simplifies complicated ones. It is certainly true for types, several distinct systems can lead confusion discomfort To address this issue, we propose hierarchical deep multi-modal network analyzes classifies end-user questions/queries then incorporates query-specific approach prediction. We refer our proposed as Hierarchical Segregation based Answering, short HQS-VQA. Our contributions three-fold, viz. firstly, segregation (QS) technique VQA-Med; secondly, integrate QS model neural generate proper answers queries related images; thirdly, study impact of Medical-VQA by comparing performance without QS. evaluate on two benchmark datasets, RAD CLEF18. Experimental results show HQS-VQA outperforms baseline models significant margins. also conduct quantitative qualitative analysis obtained discover potential causes errors their solutions.